Device, system and method for cognitive image capture

ABSTRACT

An image capture device includes a camera lens configured to capture image data, a microphone configured to receive a voice command from a user, a network interface configured to establish a network connection, a memory storing a computer program, and a processor configured to execute the computer program. The computer program is configured to identify a target subject and an image capture function included in the voice command using natural language processing (NLP). The computer program is configured to search a first database for the target subject and search a second database for the target subject, extract a visual representation of the target subject, detect the target subject in an environment viewed through camera lens, apply the image capture function to the target subject detected in the environment, and capture the image data. The image data includes the target subject with the image capture function applied to the target subject.

BACKGROUND

Exemplary embodiments of the present invention relate to a device,system and method for adaptive image capture.

Professional grade cameras with the capacity to capture high qualityimages are being sold at an increasing rate. Digital cameras, forexample, have become relatively easy to use, but also include an arrayof professional grade adjustable settings. A digital camera may allow anovice camera user to take high quality pictures by using automatedfeatures of the digital camera without substantial knowledge of how toadjust camera parameters such as shutter speed, ISO settings andaperture size. While high quality images may be captured by a noviceuser with limited understanding of the camera's settings, digitalcameras also include an array of settings which may be manually adjustedto further increase picture quality or to modify stylisticcharacteristics of an image being captured.

Digital cameras may be manually adjusted to focus on a particular objectin a landscape. Camera settings such as shutter speed, ISO setting andaperture size may be manually adjusted to improve image quality or toachieve desired artistic or stylistic characteristics in a capturedimage. However, novice camera users may lack knowledge or interest inlearning how to adjust such manual settings, and therefore many manualadjustment settings and features included with a camera may go unused.

Thus, it may be desirable to have an image capture device (e.g., digitalcamera), and a system and method for dynamic image capture with a userfriendly interface for adjusting the settings of the image capturedevice.

SUMMARY

Exemplary embodiments of the present invention provide an image capturedevice including a camera lens configured to capture image data, amicrophone configured to receive a voice command from a user, a networkinterface configured to establish a network connection, a memory storinga computer program, and a processor configured to execute the computerprogram. The computer program is configured to identify a target subjectand an image capture function included in the voice command usingnatural language processing (NLP). The computer program is configured tosearch a first database for the target subject, wherein the firstdatabase is stored on the memory. The computer program is configured tosearch a second database for the target subject when the target subjectis not found in the first database, wherein the second database islocated remote from the image capture device and is accessed via thenetwork connection, such as an Internet connection. The computer programis configured to extract a visual representation of the target subjectfrom the first database or the second database upon finding the targetsubject in the first database or the second database. The computerprogram is configured to detect the target subject in an environmentviewed through camera lens by comparing the visual representation of thetarget subject with objects appearing in the environment. The computerprogram is configured to apply the image capture function to the targetsubject detected in the environment. The computer program is configuredto capture the image data. The image data includes the target subject,and the image capture function has been applied to the target subject inthe captured image data.

According to an exemplary embodiment of the present invention the targetsubject may be a person or animal in the environment. The person oranimal may be identified in the voice command by the person's oranimal's name.

According to an exemplary embodiment of the present invention the imagecapture device may include an input interface such as, for example,touchscreen interface or a biometry interface (e.g., a fingerprintreader) configured to receive input from the user. The computer programmay be configured to log in to a social media network using credentialsprovided by the user through the touchscreen interface or any othermeans of providing credentials, like biometry. The second database maybe a collection of Internet-based data bases like the social medianetwork or public databases and databases present on nearby devices likethe users smartphone.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include searching acontact list of the social media network for the person's or animal'sname to identify the visual representation of the target subject.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include identifying facesfrom among the objects appearing in the environment, and comparing theidentified faces from the environment with faces in a contact list ofthe social media network to identify the visual representation of thetarget subject.

According to an exemplary embodiment of the present invention the targetsubject may be one of the objects appearing in the environment.

According to an exemplary embodiment of the present invention the seconddatabase may be a public image database accessible via the networkconnection.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include searching thepublic image database using a name of the target subject to identify thevisual representation of the target subject. The name of the targetsubject may be included in the voice command.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include comparing theobjects appearing in the environment with objects stored in the publicimage database to identify the visual representation of the targetsubject.

Exemplary embodiments of the present invention provide a method ofcapturing image data including receiving a voice command from a user.The voice command is received by a microphone of an image capturedevice. A target subject and an image capture function included in thevoice command is identified using natural language processing (NLP). Afirst database is searched for the target subject. The first database isstored on a memory disposed in the image capture device. A seconddatabase is searched for the target subject when the target subject isnot found in the first database. The second database is located remotefrom the image capture device and is accessed via a network connectionestablished using a network interface of the image capture device. Avisual representation of the target subject is extracted from the firstdatabase or the second database upon finding the target subject in thefirst database or the second database. The target subject is detected inan environment viewed through a camera lens of the image capture deviceby comparing the visual representation of the target subject withobjects appearing in the environment. The image capture function isapplied to the target subject detected in the environment. The imagedata is captured. The image data includes the target subject, and theimage capture function has been applied to the target subject in thecaptured image data.

According to an exemplary embodiment of the present invention the targetsubject may be a person or animal in the environment. The person may beidentified in the voice command by the person's or animal's name.

According to an exemplary embodiment of the present invention the methodof capturing image data may include logging in to a social medianetwork, by the user, via a user interface of the image capture device.The second database may be the social media network.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include searching acontact list of the social media network for the person's or animal'sname to identify the visual representation of the target subject.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include identifying facesfrom among the objects appearing in the environment, and comparing theidentified faces from the environment with faces in a contact list ofthe social media network to identify the visual representation of thetarget subject.

According to an exemplary embodiment of the present invention the targetsubject may be one of the objects appearing in the environment.

According to an exemplary embodiment of the present invention the seconddatabase may be a public image database accessible via the networkconnection.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include searching thepublic image database using a name of the target subject to identify thevisual representation of the target subject. The name of the targetsubject may be included in the voice command.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include comparing theobjects appearing in the environment with objects stored in the publicimage database to identify the visual representation of the targetsubject.

According to an exemplary embodiment of the present invention the imagecapture function may include one of a sharpness modification function, abrightness modification function, a focus modification function, amotion blur modification function, a temperature modification function,and a contract modification function.

Exemplary embodiments of the present invention provide a computerprogram product for capturing image data, the computer program productincluding a computer readable storage medium having program instructionsembodied therewith, the program instructions executable by a processorto cause the processor to perform the following. Receive a voice commandfrom a user. The voice command is received by a microphone of an imagecapture device. Identify a target subject and an image capture functionincluded in the voice command using natural language processing (NLP).Search a first database for the target subject. The first database isstored on a memory disposed in the image capture device. Search a seconddatabase for the target subject when the target subject is not found inthe first database. The second database is located remote from the imagecapture device and is accessed via a network connection establishedusing a network interface of the image capture device. Extract a visualrepresentation of the target subject from the first database or thesecond database upon finding the target subject in the first database orthe second database. Detect the target subject in an environment viewedthrough a camera lens of the image capture device by comparing thevisual representation of the target subject with objects appearing inthe environment. Apply the image capture function to the target subjectdetected in the environment. Capture the image data. The image dataincludes the target subject, and the image capture function has beenapplied to the target subject in the captured image data.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features of the present invention will become moreapparent by describing in detail exemplary embodiments thereof, withreference to the accompanying drawings, in which:

FIG. 1 illustrates an image capture device according to an exemplaryembodiment of the present invention.

FIG. 2 is a flowchart illustrating a method of capturing image dataaccording to an exemplary embodiment of the present invention.

FIG. 3 is a flowchart illustrating a method of capturing image datausing a voice command according to an exemplary embodiment of thepresent invention.

FIG. 4 is a flowchart illustrating a system and method for capturingimage data using a voice command according to an exemplary embodiment ofthe present invention.

FIG. 5 is a flowchart illustrating a camera tuning advisor system andmethod according to an exemplary embodiment of the present invention.

FIG. 6 illustrates a conceptual model of a camera according to anexemplary embodiment of the present invention.

FIG. 7 is a flowchart illustrating a camera tuning advisor system andmethod according to an exemplary embodiment of the present invention.

FIG. 8 illustrates a system and method for adaptive camera self-learningaccording to an exemplary embodiment of the present invention.

FIG. 9 illustrates an example of a computer system capable ofimplementing the methods according to exemplary embodiments of thepresent invention.

DETAILED DESCRIPTION

Although nearly all modern digital cameras can focus the lensautomatically they also allow you to focus manually instead. Manualfocus is a particularly good option with macro photography because manycameras struggle to lock onto very close subjects and the lens ends uphunting (moving in and out of focus) every time the shutter releasebutton is depressed. It may be far less frustrating to switch the camera(or the lens) to manual focus and do the job yourself. For example,accessing the manual settings of a camera may allow the user to focus ona desired person or object which is not substantially centered in thecamera's viewfinder, or which is not easily focused on by the autofocussettings of the camera. However, many camera users may lack theknowledge or confidence to access and adjust the camera's manualsettings.

Exemplary embodiments of the present invention provide a device, systemand method for an adaptive personalized social smart camera. A user ofthe camera, system and method according to exemplary embodiments of thepresent invention may interact with the camera through natural languagecommands, and/or by giving specific commands or high-level goals. Forexample, a user may provide a natural language command or a high-levelgoal by referencing entities in the scene. The camera may then react bychanging the camera's settings, identifying and tagging people, objectsor places in the screen by name or image using online social networks oronline search engines, or by automatically tuning the camera, given thelearned features acquired during its usage. The camera may also give theuser hints to improve image composition.

Exemplary embodiments of the present invention will be described morefully hereinafter with reference to the accompanying drawings. Likereference numerals may refer to like elements throughout thespecification and drawings.

FIG. 1 illustrates an image capture device according to an exemplaryembodiment of the present invention.

Referring to FIG. 1, exemplary embodiments of the present inventionprovide an image capture device 100 (e.g., a camera such as a digitalcamera) including a camera lens 101 configured to capture image data, amicrophone 102 configured to receive a voice command from a user, anetwork interface 103 configured to establish a network/Internetconnection, a memory 104 storing a computer program, and a processor 105configured to execute the computer program. The computer program isconfigured to identify a target subject and an image capture functionincluded in the voice command using natural language processing (NLP).The computer program is configured to search a first database for thetarget subject, wherein the first database is stored on the memory. Thecomputer program is configured to search a second database for thetarget subject when the target subject is not found in the firstdatabase, wherein the second database is located remote from the imagecapture device and is accessed via the network/Internet connection. Thecomputer program is configured to extract a visual representation of thetarget subject from the first database or the second database uponfinding the target subject in the first database or the second database.The computer program is configured to detect the target subject in anenvironment viewed through camera lens by comparing the visualrepresentation of the target subject with objects appearing in theenvironment. The computer program is configured to apply the imagecapture function to the target subject detected in the environment. Thecomputer program is configured to capture the image data. The image dataincludes the target subject, and the image capture function has beenapplied to the target subject in the captured image data.

The image capture device 100 according to exemplary embodiments of thepresent invention may be any electronic device that can capture images,such as digital images. For example, the image capture device 100 may bea standalone camera (e.g., a digital camera), a tablet, a smartphone, ora tablet computer. However, exemplary embodiments of the presentinvention are not limited thereto, and the image capture device 100 maybe any desired device for capturing images or pictures. That is, evenwhen a “camera” is referred to, it will be understood that the termcamera is not intended to be limiting, and a camera may be replaced witha standalone camera (e.g., a digital camera), a tablet, a smartphone, ora tablet computer, for example. Thus, the terms camera, and terms (e.g.,smartphone) referring to other image capture devices may be usedinterchangeably herein.

The image capture device 100 according to an exemplary embodiment of thepresent invention may include a user interface 106. The user interface106 may include one or more interface screens, such as a touch screenfor receiving input from the user and/or displaying image capture deviceinformation to the user. The user may interact with the user interface106 to modify the camera settings or to respond to a prompt by the imagecapture device. For example, the image capture device 100 may pose aquery to the user regarding why an image was not captured, and theuser's response may be used by a self-learning image capture device whencapturing images in the future, as discussed below in more detail.

The network interface 103 according to exemplary embodiments of thepresent invention may transmit and/or receive a WiFi, cellular or anydesired wireless signal, however, exemplary embodiments of the presentinvention are not limited thereto, and the network interface 103 mayinclude any desired network interface which may be configured toestablish an Internet connection.

According to an exemplary embodiment of the present invention the targetsubject may be a person or animal in the environment. The person may beidentified in the voice command by the person's or animal's name.

According to an exemplary embodiment of the present invention the imagecapture device may include an input interface such as, for example, atouchscreen interface or a biometric interface (e.g., a fingerprintreader) configured to receive input from the user. The computer programmay be configured to log in to a social media network using credentialsprovided by the user through the touchscreen interface. The seconddatabase may be the social media network.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include searching acontact list of the social media network for the person's or animal'sname to identify the visual representation of the target subject.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include identifying facesfrom among the objects appearing in the environment, and comparing theidentified faces from the environment with faces in a contact list ofthe social media network to identify the visual representation of thetarget subject.

According to an exemplary embodiment of the present invention the targetsubject may be one of the objects appearing in the environment.

According to an exemplary embodiment of the present invention the seconddatabase may be a public image database accessible via thenetwork/Internet connection.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include searching thepublic image database using a name of the target subject to identify thevisual representation of the target subject. The name of the targetsubject may be included in the voice command.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include comparing theobjects appearing in the environment with objects stored in the publicimage database to identify the visual representation of the targetsubject.

FIG. 2 is a flowchart illustrating a method of capturing image dataaccording to an exemplary embodiment of the present invention.

Referring to FIG. 2, a method of capturing image data includes receivinga voice command from a user at operation 201. The voice command isreceived by a microphone (e.g., the microphone 102) of an image capturedevice (e.g., image capture device 100). A target subject and an imagecapture function included in the voice command is identified usingnatural language processing (NLP) at operation 202. A first database issearched for the target subject at operation 203. The first database isstored on a memory disposed in the image capture device. A seconddatabase is searched for the target subject when the target subject isnot found in the first database at operation 204. The second database islocated remote from the image capture device and is accessed via anetwork/Internet connection established using a network interface of theimage capture device. A visual representation of the target subject isextracted from the first database or the second database upon findingthe target subject in the first database or the second database atoperation 205. The target subject is detected in an environment viewedthrough a camera lens of the image capture device by comparing thevisual representation of the target subject with objects appearing inthe environment at operation 206. The image capture function is appliedto the target subject detected in the environment at operation 207. Theimage data is captured at operation 208. The image data includes thetarget subject, and the image capture function has been applied to thetarget subject in the captured image data.

A target subject according to exemplary embodiments of the presentinvention may refer to, for example, a person, an animal, an object, aplace, or any desired combination of a person(s), object(s) and/orplace(s), as desired.

An image capture function according to an exemplary embodiment of thepresent invention may refer to a specific task or command (e.g., anatural language command spoken by the user of the image capture device)for the image capture device included in the user's voice command. Forexample, an image capture function may include a command to “pleasefocus on Maria,” “please focus on the boat,” “please increase thecontrast between the boat and water” and “please focus on the tree andleave the background de-focused.”

According to an exemplary embodiment of the present invention the targetsubject may be a person or animal in the environment. The person oranimal may be identified in the voice command by the person's oranimal's name.

According to an exemplary embodiment of the present invention the methodof capturing image data may include logging in to a social medianetwork, by the user, via a user interface (e.g., user interface 106) ofthe image capture device. The second database may be the social medianetwork. The user interface may include one or more interface screens,such as a touch screen for receiving input from the user and/ordisplaying image capture device information to the user. The user mayinteract with the user interface to modify the camera settings or torespond to a prompt by the image capture device. For example, the imagecapture device may pose a query to the user regarding why an image wasnot captured, which may be used by a self-learning image capture device,as discussed below in more detail.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include searching acontact list of the social media network for the person's or animal'sname to identify the visual representation of the target subject.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include identifying facesfrom among the objects appearing in the environment, and comparing theidentified faces from the environment with faces in a contact list ofthe social media network to identify the visual representation of thetarget subject. Database searching is described in more detail below,for example, with reference to FIGS. 4 and 8.

According to an exemplary embodiment of the present invention the targetsubject may be one of the objects appearing in the environment.

According to an exemplary embodiment of the present invention the seconddatabase may be a public image database (e.g. Google® images or Yahoo®images) accessible via the Internet connection.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include searching thepublic image database using a name of the target subject to identify thevisual representation of the target subject. The name of the targetsubject may be included in the voice command.

According to an exemplary embodiment of the present invention searchingthe second database for the target subject may include comparing theobjects appearing in the environment with objects stored in the publicimage database to identify the visual representation of the targetsubject. Database searching is described in more detail below, forexample, with reference to FIGS. 4 and 8.

According to an exemplary embodiment of the present invention the imagecapture function may include one of a sharpness modification function, abrightness modification function, a focus modification function, amotion blur modification function, a temperature modification function,a contrast modification function, or any other camera related functiontypically provided by a camera. However, the image capture function isnot limited thereto.

FIG. 3 is a flowchart illustrating a method of capturing image datausing a voice command according to an exemplary embodiment of thepresent invention.

Referring to FIG. 3, a method of capturing image data using a voicecommand may include a user starting a camera at operation 301 (e.g.,image capture device 100, which may be a digital camera), and the camerastarting and detecting the environment at operation 302. The user maypoint the camera and send a command by voice to the camera at operation303. The camera may recognize the command at operation 304 and replywith a voice at operation 306 back to the user, which may prompt theuser to point the camera differently and/or respond with a second voicecommand. When the camera recognizes a command, the camera may run thecommand at operation 305 and may then respond back to the user with thevoice reply at operation 306. A more detailed description of camera selflearning, and processes employed by a camera to adaptively recognize andrespond to a voice command are described below in more detail withreference to FIG. 8.

FIG. 4 is a flowchart illustrating a system and method for capturingimage data using a voice command according to an exemplary embodiment ofthe present invention.

Referring to FIG. 4, a system and method for capturing image data usinga voice command according to an exemplary embodiment of the presentinvention may include a start operation 301 when the user says the nameof the object (the object may be a person, for example) in theenvironment and points the camera to it at operation 302. The camera(e.g., image capture device 100, which may be a digital camera) maysearch in the camera's internal historical database at operation 303,which may be stored in an internal memory 314 of the camera. The cameramay determine if the object is found at operation 304. If the object isfound (e.g., in the camera's internal memory 314) the camera may detectthe object by similarity of image at operation 305. If the object is notfound (e.g., in the camera's internal memory 314) the camera may searchthe Internet 320 (e.g., Google® Images or Yahoo® Images) by name atoperation 306.

When the object is found in operation 304 and the camera detects theobject by similarity of image at operation 305, the camera may focus onthe object at operation 311 and the camera may query the user (e.g.,“photo shoot?”) at operation 312 to determine whether the image ascurrently focused should be taken. If the camera receives a response toshoot the photo then the camera may update the camera's internal memory314 at operation 313. If the camera receives a response to not shoot thephoto then the image capture process may end at operation 317. However,the camera may also update its internal memory when an image is nottaken at operation 313. A decision to shoot or not shoot a photo may beused for a camera self-leaning process (e.g., to better understand theuser's preferences). A more detailed description of cameraself-learning, and processes employed by a camera to adaptivelyrecognize and respond to a voice command are described below in moredetail with reference to FIGS. 4 and 8.

If the object is not found in operation 304 and the camera searches theInternet 320 by name at operation 306, then it will be determinedwhether the object is found by name on the Internet at operation 307.

If the object is found on the Internet by name in operation 307, thenthe camera may proceed from operation 305, as described above in moredetail.

If the object is not found on the Internet by name in operation 307,then the camera may search the Internet by image similarity (e.g., usingGoogle® Images or Yahoo® Images) at operation 308. It will then bedetermined whether the object is found by image similarity at operation309. If it is determined at operation 309 that the object was not foundby image similarity, then the camera may reply with a not found messageto the user at operation 310, and the process may end at operation 317.The camera may update its internal memory 314 at operation 313 toindicate that the object was not found by name or image similarity onthe Internet.

When searching the Internet by image similarity at operation 308, thecamera may search the Internet using facial recognition or objectrecognition. When searching the Internet using name recognition atoperation 306, the camera may perform a text based search for objectslabeled (e.g., objects including metadata) with a desired name. Forexample, the camera may search one or more of the user's social networkaccounts to identify people or objects that explicitly include a samename (e.g., John Smith), such as in a labeled image, or may search forobjects by object or facial recognition.

According to an exemplary embodiment of the present invention, searcheson a user's social network may include searching within the user'sfriends in the user's network. A graph search algorithm may be employedin searching the user's network. If the search is specific to the user'sfriends, the search can be accomplished by visiting each friend'sprofile and gathering their information. This can be done using thesocial network API and the user's credentials (e.g., Facebook API) orwith an HTTP crawler using, for instance, a PageRank algorithm in a caseof a public social network like Twitter.

According to an exemplary embodiment of the present invention the methodof capturing image data may include logging in to a social medianetwork, by the user, via a user interface (e.g., user interface 106) ofthe image capture device and searching one or more databases of thesocial media network. The user interface may include one or moreinterface screens, such as a touch screen for receiving input from theuser and/or displaying image capture device information to the user. Theuser may interact with the user interface to modify the camera settingsor to respond to a prompt by the image capture device. For example, theimage capture device may pose a query to the user regarding why an imagewas not captured, which may be used by a self-learning image capturedevice, as discussed below in more detail.

According to an exemplary embodiment of the present invention searchingthe database for the target subject may include searching a contact listof the social media network for the person's or animal's name toidentify the visual representation of the target subject.

According to an exemplary embodiment of the present invention searchingthe database for the target subject may include identifying faces fromamong the objects appearing in the environment, and comparing theidentified faces from the environment with faces in a contact list ofthe social media network to identify the visual representation of thetarget subject. Database searching is described in more detail below,for example, with reference to FIGS. 4 and 8.

FIG. 5 is a flowchart illustrating a camera tuning advisor system andmethod according to an exemplary embodiment of the present invention.

Referring to FIG. 5, a camera tuning advisor system and method accordingto an exemplary embodiment of the present invention may include a person501 (e.g., a user) having a goal 502 having quality features 503 withregard to an image 504. The image 504 may include image features 505which are detected by a camera 506 (e.g., image capture device 100,which may be a digital camera). The camera settings 507 may be adjustedto achieve the goal 502 indicated by the user.

The camera tuning advisor according to exemplary embodiments of thepresent invention may automatically adjust a camera's settings inresponse to a user's indicated goal (e.g., to capture an image focusedon one person with a de-focused background), or may provide advice tothe user on how to manually adjust the camera's settings to achieve thedesired goal. The camera may communicate with the user through voicecommands and/or through a user interface such as a display screen or atouchscreen (e.g., user interface 106).

A camera tuning advisor system and method according to an exemplaryembodiment of the present invention may generate a quality feature table510, which may include measurements of one or more of sharpness,lightness (e.g., brightness), focus, motion blur, temperature andcontrast. Each goal may be associated with one or more changes to theapplied camera settings. For example, a shutter speed of the camera maybe adjusted to comply with one or more rules associated with temperaturegoals (see, e.g., table 511). Temperature goals may include a desiredresult including an increase in light and/or sharpness of an image whichcorresponds with a particular shutter speed. Thus, the camera tuningadvisor may advise a user desiring a higher or lower temperature (e.g.,higher or lower lightness and/or sharpness) of an image to increase ordecrease the shutter speed because a stated temperature goal may beconverted to a shutter speed goal.

According to an exemplary embodiment of the present invention,temperature may refer to a temperature of light, which may also bereferred to as white balance. White balance may refer to a process ofremoving unrealistic color, so that objects which appear white in personare rendered white in a users photograph. White balance may take intoaccount the color temperature of a light source, which may refer to therelative warmth or coolness of white light. For example, warmer colorsmay refer to red, yellow and orange colors in a color wheel, and coolercolors may refer to green, blue and purple colors in a color wheel.

FIG. 6 illustrates a conceptual model of camera according to anexemplary embodiment of the present invention.

Referring to FIG. 6, camera settings (e.g., camera settings 507) in acamera tuning advisor according to an exemplary embodiment of thepresent invention may include a settings feature 601 having qualityfeatures 602 and impact 603. For example, a particular settings feature601 may correspond to a particular rule, as discussed above in moredetail. Application of a particular rule may result in adjustment of oneor more quality features 602 (e.g., sharpness, lightness, focus, motionblur, temperature and/or contrast) which may have a desired impact 603.For example, one setting feature for increasing temperature may increasea first quality feature (e.g., lightness) and decrease a second qualityfeature (e.g., sharpness), which may result in a higher temperatureimage being captured.

As discussed below in more detail, a plurality of setting features maybe stored in a database (e.g., features database 804, described below inmore detail). The features may be set by a manufacturer of a particularcamera or by the camera's user and may be used and/or adjusted throughcamera self-learning, as discussed below in more detail.

FIG. 7 is a flowchart illustrating a camera tuning advisor system andmethod according to an exemplary embodiment of the present invention.

Referring to FIG. 7, a camera tuning advisor system and method accordingto an exemplary embodiment of the present invention may include a userpointing a camera at a scene and sending a command with natural languageat operation 701. The natural language command may include one or moregoals, which as discussed herein, may refer to an image or photographhaving a desired characteristic being captured. For example, a user mayprovide a natural language command stating “please focus on Graham andleave the background de-focused” or “please focus and Graham and leavethe background fuzzy.” In this case, a first goal may be to focus onsomeone named Graham, and a second goal may be to defocus on abackground area in the field of the cameras viewfinder. The camera mayparse the goal(s) at operation 702, may extract image features from ascene at operation 703, and may extract quality features from the sceneat operation 704. The camera may load setting features (e.g., qualityfeatures) from a rule database for the given goal at operation 705. Theextracted and loaded quality features may be compared at operation 706.The camera may determine whether a self-tuning step should be performedat operation 707. If it is determined that a self-tuning step should beperformed, the camera may update its settings and inform the user atoperation 708. That is, the camera may automatically adjust itssettings. If the camera determines that a self-tuning step should not beperformed, the camera may generate settings adjustment advice and sendthe advice to the user at operation 709. The user may then receiveeither advice on how to update the settings or information regarding theself-tuned updated settings at operation 710.

According to an exemplary embodiment of the present invention, thenatural language commands may include tokenization (which meanssplitting the sentences into tokens and a token may be a word or apunctuation like comma), lemmatization (which means detects the stems ofthe words), part-of-speech tagging (which means detecting the part ofspeech of each token) or named-entity recognition (which meansidentifying the class of the entities mentioned in the token). These NLPtasks can be achieved using a rule-based algorithm or supervised orunsupervised machine learning algorithms.

According to an exemplary embodiment of the present invention,extracting quality features from the scene at operation 704 and/orcomparison of the quality features at operation 706 may includeperforming an image recognition procedure. For example, the SIFTalgorithm, the SURF algorithm and deep neural network based algorithmsmay be employed. For quality feature classification, established methodsknown to those skilled in the art may be used for obtaining the desiredmetrics. For example, for overall brightness a histogram may be used.For sharpness, a method may be used employing the fast Fourier transform(FFT) or the variance of pixel values of a region of interest within theimage or the full image to determine brightness.

According to an exemplary embodiment of the present invention,generating settings adjustment advice and sending the advice to the userat operation 709 may employ one or more planning algorithms. When thegoal and the means employed by the camera to achieve the goal areformalized, the planning algorithm can compute the sequence of steps toachieve the goal or recognize that the stated goal cannot be achieved.

According to an exemplary embodiment of the present invention, parsingthe user's goal(s) at operation 702 may include the camera automaticallyidentifying what the user is trying to do without receiving or withoutunderstanding a user's stated natural language command. For example, thecamera may automatically detect that a user is attempting to focus on aparticular object (e.g., a tree) based on the user attempting to focuson such an object, even if an associated natural language goal (e.g.,“focus on the tree only”) is not understood. Thus, a goal may be parsedeven when a natural language command is either not understood or is onlypartially understood by the camera.

FIG. 8 illustrates a system and method for adaptive camera self-learningaccording to an exemplary embodiment of the present invention.

Referring to FIG. 8, a system and method for adaptive cameraself-learning according to an exemplary embodiment of the presentinvention may include a camera learning module and a camera self-tuningmodule which may access features for adjusting camera settings based onphotographs and features from photographs captured by a user in thepast.

According to an exemplary embodiment of the present invention, thelearning module may learn a user's preferred camera settings fordifferent picture taking contexts. To implement camera self-learning, auser may adjust camera settings at operation 801. The setting may beadjusted manually, through natural language commands, and/or based onrecommendations from the camera tuning advisor, as described herein. Theuser may then shoot the photo at operation 802. The camera may extractone or more features of the photo and save features and settingsassociated with the photo at operation 803. The photos includingassociated features and settings may be stored in a features database804 (e.g., in the camera's internal memory). The features database 804may access or may include camera features set by one or more cameramanufacturers, which may be used to adjust the camera's settings, asdesired. The user may manually identify photographs which have beensuccessfully captured and achieve one or more of the user's goals. Forexample, a user may identify one or more photographs focusing on aperson of interest while leaving the background de-focused. For example,the user may identify photographs that are considered to haveappropriate temperature settings so that in the future the camera canrefer to such an image in determining a user's desired temperaturesettings. Thus, the camera may learn the user's preferred settings indifferent contexts and with regard to different image capture goals bythe user identifying which images include the desired features, impactor other characteristics.

According to an exemplary embodiment of the present invention, cameralearning can be accomplished by using a hand-crafted or learning featureextraction mechanism. The hand-crafted feature extraction mechanism maybe performed prior to the camera usage, while the learning featureextraction mechanism may be used during camera usage and may beautomatic. In this case, convolutional neural networks (aka deeplearning) may be used.

According to an exemplary embodiment of the present invention, thecamera self-tuning module may perform the following steps. The cameramay be focused based on a user's request (e.g., a natural languagerequest) or previous settings at operation 805. That is, the camera maybe initially manually adjusted, or may be adjusted according to naturallanguage commands and/or suggestions received from the camera tuningadvisor described herein. The camera may extract features from thescenery at operation 806 (e.g., the scenery in the camera's viewfinder).The camera may then search for settings at operation 807 that can bestbe applied to achieve one or more of the user's goals. That is, thecamera may access one or more learned features, and may use the accessedlearned features to adjust the camera's settings.

According to exemplary embodiments of the present invention, cameraself-tuning may be performed automatically by the camera and/or mayinclude the camera providing setting adjustment advice to the user viathe camera tuning advisor described herein. The camera may determinewhether desired settings (e.g., learned settings) are found. If desiredsettings are not found, the camera may maintain the current settings andthe camera tuning may end at operation 812. If desired settings areidentified, the camera may determine whether the camera will self-tuneat operation 809. If the camera determines to self-tune then the cameratunes its settings at operation 811 and the tuning process may end atoperation 812. If the camera determines not to self-tune, it may sendhints to the user about the settings at operation 810. That is, thecamera tuning advisor, as described herein, may provide additionaladvice regarding improved camera settings to the user. Thus, thelearning module may provide camera settings which may be used to furtherimprove the camera's settings to come in line with a user's identifiedpreferred camera settings to achieve one or more desired image capturegoals.

According to an exemplary embodiment of the present invention thefeatures database 804 may be stored in and accessed from the cloud(e.g., through an Internet connection) or may be stored in the camera'sinternal memory.

According to an exemplary embodiment of the present invention, thefeatures database 804 may tabulate and store one or more tables (e.g.,table 820), which include camera settings associated with one or morefeatures. For example, features may include settings associated with anumber of persons in a scene, scenes with sun and no persons, cloudyscenes with people, indoors scenes with no persons, and indoor sceneswith people. Thus, features may be associated with one or more picturetaking contexts. The tables (e.g., table 820) may represent a user'sidentified camera settings when a particular image is captured (e.g., animage with cool temperatures in an indoor setting with people vs. animage with warm temperatures in an indoor setting with people). That is,a user may identify ideal or near ideal pictures which have beenpreviously captured using particular camera settings, and the settingsassociated with the ideal or near ideal pictures may be tabulated andstored in the features database 804. Thus, the ideal or near idealsettings for different contexts or to achieve different goals may belearned, and the associated settings accessed for camera self-tuning(either automated or via recommended settings from the camera tuningadvisor).

According to an exemplary embodiment of the present invention, thefeatures database 804 may include a classifier, which may be trained tolearn a user's ideal or near ideal settings to achieve different imagecapture goals in different contexts. The classifier may be trainedusing, for example, a features table (e.g., table 820).

According to an exemplary embodiment of the present invention, cameraself-learning may be used to store settings for more than one user of aparticular camera. For example, as illustrated in table 830, featuressuch as sharpness, lightness, focus, motion blur, temperature andcontrast may be mapped to a numeric vector, and different numericvectors may be associated with one or more users. A camera may identifya particular user is currently using a camera and use an image featurevector table (e.g., table 830) to apply a particular set of features toa camera that are specific to that user. For example, a first user Bobmay prefer cooler temperature pictures in all contexts, while a seconduser Alice may prefer warmer temperature pictures in all contexts. Thus,the camera learning module may be employed to learn the particularsettings of more than one user of a same camera.

FIG. 9 illustrates an example of a computer system capable ofimplementing the methods according to exemplary embodiments of thepresent invention. The system and method of the present disclosure maybe implemented in the form of a software application running on acomputer system, for example, a mainframe, personal computer (PC),handheld computer, server, etc. The software application may be storedon a recording media locally accessible by the computer system andaccessible via a hard wired or wireless connection to a network, forexample, a local area network, or the Internet.

The computer system referred to generally as system 900 may include, forexample, a central processing unit (CPU) 901, random access memory (RAM)904, a printer interface 910, a display unit 911, a local area network(LAN) data transmission controller 905, a LAN interface 906, a networkcontroller 903, an internal bus 902, and one or more input devices 909,for example, a keyboard, mouse etc. As shown, the system 900 may beconnected to a data storage device, for example, a hard disk, 908 via alink 907.

Exemplary embodiments of the present invention provide a computerprogram product for capturing image data, the computer program productincluding a computer readable storage medium having program instructionsembodied therewith, the program instructions executable by a processor(e.g., CPU 901) to cause the processor (e.g., CPU 901) to perform thefollowing. Receive a voice command from a user. The voice command isreceived by a microphone of an image capture device. Identify a targetsubject and an image capture function included in the voice commandusing natural language processing (NLP). Search a first database for thetarget subject. The first database is stored on a memory disposed in theimage capture device. Search a second database for the target subjectwhen the target subject is not found in the first database. The seconddatabase is located remote from the image capture device and is accessedvia an Internet connection established using a network interface of theimage capture device. Extract a visual representation of the targetsubject from the first database or the second database upon finding thetarget subject in the first database or the second database. Detect thetarget subject in an environment viewed through a camera lens of theimage capture device by comparing the visual representation of thetarget subject with objects appearing in the environment. Apply theimage capture function to the target subject detected in theenvironment. Capture the image data. The image data includes the targetsubject, and the image capture function has been applied to the targetsubject in the captured image data.

The descriptions of the various exemplary embodiments of the presentinvention have been presented for purposes of illustration, but are notintended to be exhaustive or limited to the exemplary embodimentsdisclosed. Many modifications and variations will be apparent to thoseof ordinary skill in the art without departing from the scope and spiritof the described exemplary embodiments. The terminology used herein waschosen to best explain the principles of the exemplary embodiments, orto enable others of ordinary skill in the art to understand exemplaryembodiments described herein.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire. Computer readable program instructions described hereincan be downloaded to respective computing/processing devices from acomputer readable storage medium or to an external computer or externalstorage device via a network, for example, the Internet, a local areanetwork, a wide area network and/or a wireless network. The network maycomprise copper transmission cables, optical transmission fibers,wireless transmission, routers, firewalls, switches, gateway computersand/or edge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowcharts and/or block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousexemplary embodiments of the invention. In this regard, each block inthe flowchart or block diagrams may represent a module, segment, orportion of instructions, which comprises one or more executableinstructions for implementing the specified logical function(s). Inalternative implementations, the functions noted in the block may occurout of the order noted in the figures. For example, two blocks shown insuccession may, in fact, be executed substantially concurrently, or theblocks may sometimes be executed in the reverse order, depending uponthe functionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While the present invention has been particularly shown and describedwith reference to exemplary embodiments thereof, it will be understoodby those of ordinary skill in the art that various changes in form anddetail may be made therein without departing from the spirit and scopeof the present invention as defined by the following claims.

The invention claimed is:
 1. An image capture device, comprising: acamera lens configured to capture image data; a microphone configured toreceive a voice command from a user; a network interface configured toestablish a network connection; a memory storing a computer program; anda processor configured to execute the computer program, wherein thecomputer program is configured to: identify a target subject and animage capture function included in the voice command using naturallanguage processing (NLP); search a first database for the targetsubject, wherein the first database is stored on the memory; search asecond database for the target subject when the target subject is notfound in the first database, wherein the second database is locatedremote from the image capture device and is accessed via the networkconnection, and wherein searching the second database for the targetsubject comprises comparing a visual representation of objects appearingin an environment viewed through the camera lens with visualrepresentations of objects stored in the second database to identify avisual representation of the target subject; extract the visualrepresentation of the target subject from the first database or thesecond database upon finding the target subject in the first database orthe second database; detect the target subject in the environment viewedthrough the camera lens by comparing the visual representation of thetarget subject with objects appearing in the environment; apply theimage capture function to the target subject detected in theenvironment; and capture the image data, wherein the image datacomprises the target subject, and the image capture function has beenapplied to the target subject in the captured image data.
 2. The imagecapture device of claim 1, wherein the target subject is a person oranimal in the environment, and the person or animal is identified in thevoice command by the person's or animal's name.
 3. The image capturedevice of claim 2, further comprising: an input interface configured toreceive input from the user, wherein the computer program is configuredto log in to a social media network using credentials provided by theuser through the input interface, wherein the second database is thesocial media network.
 4. The image capture device of claim 3, whereinsearching the second database for the target subject comprises searchinga contact list of the social media network for the person's name toidentify the visual representation of the target subject.
 5. The imagecapture device of claim 3, wherein searching the second database for thetarget subject comprises: identifying faces from among the objectsappearing in the environment; and comparing the identified faces fromthe environment with faces in a contact list of the social media networkto identify the visual representation of the target subject.
 6. Theimage capture device of claim 1, wherein the target subject is one ofthe objects appearing in the environment.
 7. The image capture device ofclaim 6, wherein the second database is a public image databaseaccessible via the network connection.
 8. The image capture device ofclaim 7, wherein searching the second database for the target subjectcomprises searching the public image database using a name of the targetsubject to identify the visual representation of the target subject,wherein the name of the target subject is included in the voice command.9. A method of capturing image data, comprising: receiving a voicecommand from a user, wherein the voice command is received by amicrophone of an image capture device; identifying a target subject andan image capture function included in the voice command using naturallanguage processing (NLP); searching a first database for the targetsubject, wherein the first database is stored on a memory disposed inthe image capture device; searching a second database for the targetsubject when the target subject is not found in the first database,wherein the second database is located remote from the image capturedevice and is accessed via a network connection established using anetwork interface of the image capture device; extracting a visualrepresentation of the target subject from the first database or thesecond database upon finding the target subject in the first database orthe second database; detecting the target subject in an environmentviewed through a camera lens of the image capture device by comparingthe visual representation of the target subject with objects appearingin the environment; applying the image capture function to the targetsubject detected in the environment; capturing the image data, whereinthe image data comprises the target subject, and the image capturefunction has been applied to the target subject in the captured imagedata, wherein the target subject is a person or animal in theenvironment, and the person or animal is identified in the voice commandby the person's or animal's name; and logging in to a social medianetwork, by the user, via a user interface of the image capture device,wherein the second database is the social media network, whereinsearching the second database for the target subject comprises:identifying faces from among the objects appearing in the environment;and comparing the identified faces from the environment with faces in acontact list of the social media network to identify the visualrepresentation of the target subject.
 10. The method of claim 9, whereinthe target subject is a person or animal in the environment, and theperson or animal is identified in the voice command by the person's oranimal's name.
 11. The method of claim 9, wherein the target subject isone of the objects appearing in the environment.
 12. The method of claim11, wherein the second database is a public image database accessiblevia the network connection.
 13. The method of claim 12, whereinsearching the second database for the target subject comprises searchingthe public image database using a name of the target subject to identifythe visual representation of the target subject, wherein the name of thetarget subject is included in the voice command.
 14. The method of claim12, wherein searching the second database for the target subjectcomprises comparing the objects appearing in the environment withobjects stored in the public image database to identify the visualrepresentation of the target subject.
 15. The method of claim 9, whereinthe image capture function comprises one of a sharpness modificationfunction, a brightness modification function, a focus modificationfunction, a motion blur modification function, a temperaturemodification function, and a contrast modification function.
 16. Acomputer program product for capturing image data, the computer programproduct comprising a non-transitory computer readable storage mediumhaving program instructions embodied therewith, the program instructionsexecutable by a processor to cause the processor to: receive a voicecommand from a user, wherein the voice command is received by amicrophone of an image capture device; identify a target subject and animage capture function included in the voice command using naturallanguage processing (NLP); search a first database for the targetsubject, wherein the first database is stored on a memory disposed inthe image capture device; search a second database for the targetsubject when the target subject is not found in the first database,wherein the second database is located remote from the image capturedevice and is accessed via a network connection established using anetwork interface of the image capture device, and wherein searching thesecond database for the target subject comprises comparing a visualrepresentation of objects appearing in an environment viewed through acamera lens with visual representations of objects stored in the seconddatabase to identify a visual representation of the target subject;extract the visual representation of the target subject from the firstdatabase or the second database upon finding the target subject in thefirst database or the second database; detect the target subject in theenvironment viewed through the camera lens of the image capture deviceby comparing the visual representation of the target subject withobjects appearing in the environment; apply the image capture functionto the target subject detected in the environment; and capture the imagedata, wherein the image data comprises the target subject, and the imagecapture function has been applied to the target subject in the capturedimage data.